{"cells": [{"cell_type": "markdown", "metadata": {"tags": ["module-dsml"]}, "source": ["(Other_Activation_Functions)=\n", "# Chapter 16 -- Other Activation Functions\n", "[Data Science and Machine Learning for Geoscientists](module-dsml) \n"]}, {"cell_type": "markdown", "metadata": {}, "source": ["\n", "The other solution for the vanishing gradient is to use other activation functions. We like the old activation function sigmoid $\\sigma(h)$ because first, it returns $0.5$ when $h=0$ (i.e. $\\sigma(0)$) and second, it gives a higher probability when the input value is positive and vice versa. This makes it the perfect activation function for predicting the probability. However, the vanishing gradient is a major problem we cannot ignore. In DNN (Deep Neural Network), fortunately, we can use the sigmoid function for only the output layer, and use other activation functions for hidden layers. Here are some alternatives for the activation function that do not share the same vanishing gradient problem."]}, {"cell_type": "markdown", "metadata": {}, "source": ["$$\n", "tanh(x)=\\frac{e^h-e^{-h}}{e^h+e^{-h}}\n", "$$ (eq16_1)"]}, {"cell_type": "markdown", "metadata": {}, "source": ["\n", "where $h=w*x+b$."]}, {"cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [{"data": {"text/html": ["\n", "\n"], "text/plain": [""]}, "execution_count": 4, "metadata": {}, "output_type": "execute_result"}], "source": ["# make the figure be plotted at the centre\n", "from IPython.core.display import HTML\n", "HTML(\"\"\"\n", "\n", "\"\"\")"]}, {"cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [{"data": {"image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEYCAYAAACZaxt6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXhd5Xnu/+8tyfJsPA/YFjbGzBhDhBNIUkYTQygmJCRAkgNpckib0JycDA0k/WWAkEObnpC2IaWUDDSkzCX4AMGYqWHGBoxHwAMYy5ZleR41P78/9hLdFpItWdpae0v357r2tdfwrrWf7UG31ruGVxGBmZlZZxSlXYCZmRU+h4mZmXWaw8TMzDrNYWJmZp3mMDEzs05zmJiZWac5TCxvSCqTtEtScdq1FCJJ/SSFpAntbF8k6feStkn6U67ra/HZT0r6THd+puWWw8S6naR3JO1NgqP5dWhEvBsRgyKiMQ9qLJV0X1JrSDqjndtNlVQj6Y421t+S9Z3rJNVnzf+xS7/EgZ0NnAqMi4g/y9WHSLpR0m3ZyyLirIi4O1efad3PYWJp+fMkOJpf63P5YZJKDmKzZ4HPARs6sM3NwPy2VkbEXzZ/Z+AnwN1ZfwbnHUSNnXEYsDoi9nbz51oP5DCxvCFpUnIUUJLMT5b0J0k7JT0u6ebm3/glnSGposX270g6J5n+YXJkcYekHcCVSbfONZJWSdos6R5Jw1urJSLqIuLnEfEs0K4jJUmXAtuAJzrxZ1Ai6X5JVUn301OSjspaf5ekn0uam/y5PCfpsBa7OS/5jlsl3dTG53wF+AVwRnJU9F1Jfynp8aw2+3SbHeizJZ2YdF9tlbRB0jclXQR8A7gi+ZyXk7YvSvpcMl0s6UeS3k2+968lDU7WHS2pQdIXJFVIqpb07YP987XccZhYPvsP4GVgBPBD4PMd3H42cB8wFPg98DXgIuB04FBgK5kjiU6TNAS4DvhmF+xuDjAFGAu8AdzeYv3lwLXAcKAS+FGL9ecBJwEnA19orYsuIn4JfB14Ojkq+kk7a2v1syUNAx4H/jOp+0jgTxHxB+BnwO3J58xoZZ9fBj4NfBSYCoxOtmlWDJQDRwDnAzdIOryd9Vo3cZhYWv6Q/Oa9TdIfWq6UVAacAnw/OUp4lswP2Y54ISL+EBFNSVfOl4HvRURFRNSSCahPHWQXWEvXA7+KiLWd2UlENETE7RGxKyJqyPywniGpX1azeyLi1YioJxO401vs5icRsSMi3gb+1Mr6zmjrsy8CVkbELyKiNvn8Nrv7Wvgs8NOIWBMRO4DvAZ+VpKw2P4iImmSfbwDTuuj7WBfpiv9EZgfjooh4fD/rDwW2RMSerGVrgYkd+IyWP9gPAx6Q1JS1rBEYA6zrwH73IWk6cA6Zo4FOSYLtRuBiYCTQBIjM0VlzjdnncPYAg1rs5kDrO6OtfU8EVh3kPg8F1mTNrwH6kzn6AWiMiE1tfK7lCYeJ5atKYLikAVmBkh0ku4EBzTPKXE48qsU+Wj4Sey3wFxHxXBfXegYwCXg3+WV6EFAs6diIOLmD+/oCcC5wJvAumaCrJBMoubbPnymZ7qr2Wkume601B3o0+XoyQd+sDNgLbOH9f6eWp9zNZXkpItYAC4AfJpfpngr8eVaTt4B+kj4uqQ/wt0DfA+z2FjL97YcBSBolaXZbjSX1zepeKk1OSLf2Q/1WMuc4pievW4CHgY8d8Iu+32CgBtgMDAR+fBD7OFgLgZMkHSdpAPD9Dmz7B+AISX+V/H0NkXRKsq4KmNzGnx3AncC3lLnPaDCZ7/wf4fExCorDxPLZZ8ncB7GZzA+Yu4FagIjYDnwFuI1M989uoKL13bznH8mcd3lM0k7gReCD+2n/JpnfkMcDc5Pp5iD6bvN9IRGxJyI2NL+AXUBNRFR3+BvDr4BqMt1Ji8lcntwtImIx8PfAM2TOSzzdgW23AjOBS4GNZP7sPpKsvovMEc8WSc+3svm/kDlx/zyZrrItZK4AswIih78VCkl3A29ExA/SrsXM9uUjE8tbkk6RNCW5P2QWmUt933fll5mlzyfgLZ+NJdP9MYJMF9ZfRcRr6ZZkZq1xN5eZmXWau7nMzKzTem0318iRI2PSpElpl2FmVlBeeeWVTRHxvvt/em2YTJo0iQULFqRdhplZQZG0prXl7uYyM7NOc5iYmVmnOUzMzKzTHCZmZtZpDhMzM+u0vAmTZKjOjZKWtLFekv5J0kpJiySdnLXuCkkrktcV3Ve1mZlBHoUJ8Ftg1n7Wn0dmSM+pwFVknjRKMob3D8g8/XUG8INkCFEzM+smeXOfSUT8SdKk/TSZDfx7MsbBi5KGShpHZmCieRGxBUDSPDKhdGduKzazQhYR1DU2UdvQRF3Wq76xibrGJhoag/rGJhqagsamzHRjMt0U8d7yCN5blnnx3ntEZn32PGTmIzKjhjU/0SoIsp9u1dy2uR0t2tJi2fu+375fdp91V5w2iRGDDjT8T8fkTZi0w3j2HYa1IlnW1vL3kXQVmaMaysrKclOlmXWL2oZGqnfWsnlXHZt317Jldz3b9tSxY2892/fWs7OmgZ21DeyqaWB3XQN76hrZW9fI3vrMe01DY5s/iHui7KHJLpw+vleHSWujtMV+lr9/YcStZEbFo7y8vBf9MzIrPLUNjazZvIfV1bt5Z/NuKrbuYd3WvazfVkPVzhq27alvdTsJhvTrw+B+JQzqm3kNG1DKhGHF9O9TwoDSYvqXFtOvpIi+fYrpW1JEaUkRpcWZ9z7FzS/Rp7iIkiJRUlxEcZEoKRLFWa8iiSLx39NFybwEAtHcLjONyExLKKlVaJ8f9M3T2cszbfXedHY7stalqZDCpIJ9xwCfQGbs6AoyXV3Zy5/utqrMrNO2761n4dptLK7YxvLKnSzfsIN3Nu2mKetXvkP692H80P5MHD6AGZOHM2ZIX0YP7seIQaUMG1jKiIGlDO1fyuB+JRQVpf/DtbcppDCZA1wt6S4yJ9u3R0SlpLnAT7JOup8LXJtWkWZ2YNv31PPC6k08s2ITL7+9hRUbd723rmz4AI4ZN5gLThjHlNGDmDxyIJNGDmRIvz4pVmwHkjdhIulOMkcYIyVVkLlCqw9ARNwCPAKcD6wE9gBfSNZtkXQ9MD/Z1XXNJ+PNLH+s27aXR5ds4NEllbyyZitNAQNLizll8nAuPPFQTj5sGNMmHMJgh0ZB6rWDY5WXl4efGmyWW3vqGnh4USV3z1/LgjVbATh67GDOPW4sH506kukTh9KnOJ/uULADkfRKRJS3XJ43RyZm1nNUbN3Dbc+8zf2vVLCztoHDRw3k2x87io+fMI5JIwemXZ7lgMPEzLrMqupd/PKpVTy4cB0AF0wbx2UzypgxeXheXHFkueMwMbNO27q7jp8//hZ3vPQufYrF5089jP/50cM5dGj/tEuzbuIwMbOD1tQU/O7FNfxs3lvsrKnn8g+W8fVzjmRkF98QZ/nPYWJmB2Xtlj18697XeentLXzkiJH8fxccy1FjB6ddlqXEYWJmHRIR3PtKBT+asxRJ/P2npnHJByb4nEgv5zAxs3ara2jiB3OWcufL7/Khw4fzD5ecyIRhA9Iuy/KAw8TM2mXTrlq+cservPzOFr5yxhS+ee5RFPuxJZZwmJjZAb29aTefu+0lNu2q5R8vnc7s6a0+mNt6MYeJme3Xyo07uezfXqKxKbj3L09l2oShaZdkechhYmZtWl65g8/d9hJFReLuqz7E1DG+Wsta54fimFmrMkckL9KnuMhBYgfkIxMze5+NO2u44tfzKSkq4u4vf4jDRvh5WrZ/PjIxs33srm3gi79dwJbddfzmylMcJNYuDhMze09jU/C1O19j6frt/OLykzhhwiFpl2QFwt1cZvaef3z8LZ54YyPXX3Q8Zx8zJu1yrIDkzZGJpFmS3pS0UtI1ray/SdLC5PWWpG1Z6xqz1s3p3srNeoZnV2zin59aySUfmMDnP3RY2uVYgcmLIxNJxcDNwEygApgvaU5ELGtuExH/O6v9XwMnZe1ib0RM7656zXqajTtr+PrdC5kyahA/mn1c2uVYAcqXI5MZwMqIWB0RdcBdwOz9tL8MuLNbKjPr4Rqbgq/ftZCdNfXcfPnJDCjNi98xrcDkS5iMB9ZmzVcky95H0mHAZODJrMX9JC2Q9KKki9r6EElXJe0WVFdXd0XdZgXvN8+9zfOrNvOjC4/zI+TtoOVLmLT2tLhoo+2lwH0R0Zi1rCwZ4P5y4OeSprS2YUTcGhHlEVE+atSozlVs1gOs3bKH//vYW5x99Gg+c8rEtMuxApYvYVIBZP9LngCsb6PtpbTo4oqI9cn7auBp9j2fYmatiAi++8BiigTXX3S8xyOxTsmXMJkPTJU0WVIpmcB431VZko4ChgEvZC0bJqlvMj0S+DCwrOW2ZravB15bxzMrNvGd8472WO3WaXlxpi0iGiRdDcwFioFfR8RSSdcBCyKiOVguA+6KiOwusGOAf5XURCYcb8y+CszM3m/zrlquf2gZJ5cN5XMf9GXA1nl5ESYAEfEI8EiLZd9vMf/DVrZ7Hjghp8WZ9TD/8Nhb7Kxp4O8+OY0iD3BlXSBfurnMrJu8VbWTu+e/y+dPPcxPArYu4zAx62V+8shyBvUt4WtnTU27FOtBHCZmvcgzK6p5+s1q/vqsqQwbWJp2OdaDOEzMeonGpuCGh5czYVh//sdpPuluXcthYtZLPPDaOt7YsJPvzDqaviXFaZdjPYzDxKwXaGhs4p+fXMFxhw7hgmnj0i7HeiCHiVkv8P8WrWfN5j187eypvtPdcsJhYtbDNTYF//zkSo4eO5iZHvDKcsRhYtbDPbK4ktXVu7n6rCN8g6LljMPErAdragp+8eRKpowayHnH+1yJ5Y7DxKwHe2xZFW9W7eSvz5pKsY9KLIccJmY92L89s5qy4QN8BZflnMPErId6fe02XlmzlStPm0RJsf+rW275X5hZD/Wb595mUN8SLimfkHYp1gs4TMx6oKodNTy0qJJLyicwuF+ftMuxXiBvwkTSLElvSlop6ZpW1l8pqVrSwuT1pax1V0hakbyu6N7KzfLPHS+uoTGCK0+blHYp1kvkxeBYkoqBm4GZZMaDny9pTisjJt4dEVe32HY48AOgHAjglWTbrd1Qulneqalv5PcvvcvZR4/hsBED0y7Heol8OTKZAayMiNURUQfcBcxu57YfA+ZFxJYkQOYBs3JUp1nem7NwPVt21/EXH56UdinWi+RLmIwH1mbNVyTLWvqkpEWS7pM0sYPbmvUKd7y0hiPHDOLUKSPSLsV6kXwJk9bupooW8/8PmBQR04DHgds7sG2moXSVpAWSFlRXVx90sWb5aun67Syq2M5lM8r8QEfrVvkSJhXAxKz5CcD67AYRsTkiapPZfwM+0N5ts/Zxa0SUR0T5qFGjuqRws3xy18trKS0p4hMn+eDcule+hMl8YKqkyZJKgUuBOdkNJGXfwnshsDyZngucK2mYpGHAuckys15lb10jf1i4jvOPH8vQAR6S17pXXlzNFRENkq4mEwLFwK8jYqmk64AFETEH+JqkC4EGYAtwZbLtFknXkwkkgOsiYku3fwmzlD2yuJKdNQ1cOqMs7VKsF1JEq6cXerzy8vJYsGBB2mWYdZlLbnmeTbvqePKbp/t8ieWMpFciorzl8nzp5jKzTli5cSfz39nKZ06Z6CCxVDhMzHqAu+evpaRIfPJkP4fL0uEwMStwDY1NPPDaes4+ZjSjBvdNuxzrpRwmZgXuuVWb2bSrlk+c5KMSS4/DxKzAPfBqBUP6lXDm0b53ytLjMDErYLtrG5i7tIqPTzuUviXFaZdjvZjDxKyAzV26gb31jVx8su94t3Q5TMwK2AOvrWPCsP58oGxY2qVYL+cwMStQG3fU8NzKTXzipPEUFfneEkuXw8SsQM15fT1NAbOnu4vL0ucwMStQf1i4jhPGH8IRowelXYqZw8SsEL2zaTdL1u3gwhMPTbsUM8BhYlaQHl5cCcD508YdoKVZ93CYmBWghxZVcnLZUMYP7Z92KWaAw8Ss4Kyq3sXyyh18fJq7uCx/OEzMCszDizJdXB8/wV1clj/yJkwkzZL0pqSVkq5pZf03JC2TtEjSE5IOy1rXKGlh8prTcluznuThRZWcMmkYYw/pl3YpZu/JizCRVAzcDJwHHAtcJunYFs1eA8ojYhpwH/D3Wev2RsT05HVhtxRtloIVVTt5s2onF7iLy/JMXoQJMANYGRGrI6IOuAuYnd0gIp6KiD3J7IuAn7dtvc5DiyqR4Lzjx6Zditk+8iVMxgNrs+YrkmVt+SLwx6z5fpIWSHpR0kVtbSTpqqTdgurq6s5VbJaChxdXMmPScEYPcReX5Zd8CZPWHiwUrTaUPgeUAz/NWlyWDHB/OfBzSVNa2zYibo2I8ogoHzXKYz9YYVm5cScrN+7ifJ94tzyUL2FSAUzMmp8ArG/ZSNI5wPeACyOitnl5RKxP3lcDTwMn5bJYszTMXVoFwMeOcxeX5Z98CZP5wFRJkyWVApcC+1yVJekk4F/JBMnGrOXDJPVNpkcCHwaWdVvlZt3k0SUbOKlsqK/isryUF2ESEQ3A1cBcYDlwT0QslXSdpOars34KDALubXEJ8DHAAkmvA08BN0aEw8R6lIqte1i8bjuzfFRieaok7QKaRcQjwCMtln0/a/qcNrZ7Hjght9WZpctdXJbv8uLIxMz279EllRw9djCTRg5MuxSzVjlMzPLcxp01LFizlfOO91Vclr8cJmZ5bt6yKiJglm9UtDzW4TCRNDB5/ImZdYNHl2xg8siBHDnGIypa/jpgmEgqknS5pIclbQTeAColLZX0U0lTc1+mWe+0fW89L6zazLnHjUFq7d5es/zQniOTp4ApwLXA2IiYGBGjgY+SeUbWjcld6WbWxZ5+cyMNTeGruCzvtefS4HMior7lwojYAtwP3C+pT5dXZmY8tqyKUYP7Mn3C0LRLMduvA4ZJc5BIOgv4LLANWAIsApZERG1rYWNmnVPb0MjTb2zkwunjKSpyF5flt47ctHgH8NVkm2nARcBxwBE5qMus13t+1WZ21zVy7rFj0i7F7IA6EiYrI+KBZPreXBRjZv9t3rIqBpYWc+qUEWmXYnZA7bma698lfR14QdI3u6Ems16vqSmYt6yKM44aTb8+vhLf8l97rua6PWk3Fvi8pDWS5ki6XtIluS3PrHdaWLGN6p21zHQXlxWI9pyAfwJ4onleUgmZcdpPBD6Iu7zMuty8ZVWUFIkzjxqddilm7XLAMJGkiHhv1MPkcfGLktfvWmtjZp3z2NINfOjwERwywFfdW2Fo102Lkv5aUln2Qkmlks6SdDtwRW7KM+t9VlXvYlX1bndxWUFpz9Vcs4C/AO6UdDiwFehPJogeA26KiIW5K9Gsd3l8WWbsknMcJlZADnhkEhE1EfHLiPgwUAacDZwUEYdFxP/sqiCRNEvSm5JWSrqmlfV9Jd2drH9J0qSsddcmy9+U9LGuqMcsLfOWVXHcoUMYP7R/2qWYtVu7nxqc3AF/C/At4BOSPtA89npnJU8hvhk4j8zJ/cskHdui2ReBrRFxBHAT8HfJtseSGTP+ODJHUb/0U42tUG3aVcsr7251F5cVnI48gv4O4CEyD3c8HPg+sLSL6phB5qbI1RFRB9wFzG7RZjaZy5QB7gPOVuYxqrOBu5LHurwNrEz2Z1Zwnly+kQgcJlZw8uUO+PHA2qz5CjKXHbfaJiIaJG0HRiTLX2yx7fjWPkTSVcBVAGVlZa01MUvVY8uqGD+0P8eOG5J2KWYdki93wLf2FLuWlxq31aY922YWRtwaEeURUT5q1KgOlmiWW3vrGnl2ZTUzj/XYJVZ48uUO+ApgYtb8BGB9W22SGycPAba0c1uzvPfMimpq6pvcxWUFKV/ugJ8PTJU0GVhH5oT65S3azCFzP8sLwKeAJyMiJM0B/kPSz4BDganAy11Qk1m3mresiiH9SpgxeXjapZh1WLvPmbQxnsk9XVFEcg7kamAuUAz8OiKWSroOWBARc4BfAb+TtJLMEcmlybZLJd0DLAMagK9GRGNX1GXWXRqbgiff2MiZR4+mT3FHrosxyw95M55JRDwCPNJi2fezpmuAVrvVIuIG4IauqMMsDa++u5XNu+vcxWUFK1+u5jLr1eYtq6K0uIjTj/SFIVaYOnI8/V+S/rd8mYlZl4oI5i7dwKlTRjC4nx/saIWpI2FyHPBXQKWkhyXd4PFMzDpvxcZdrNm8h3OPcxeXFa52d3NFxMUAkvqTCZbj8XgmZp322NINAMw8xmFihasj50wAiIi9wILkZWadNG9ZFdMnDmX0kH5pl2J20DryoMcRkv5K0hckzUiOUMysEyq37+X1iu3u4rKC15FzJg8Ao4CfAD8Ftkt6IydVmfUSzWOXnHvs2JQrMeucjnRzDY6I6yRdHBGnS/okXXSPiVlv9diyKg4fOZAjRg9KuxSzTunIkUlN8l4rqX9E3A+cn4OazHqFHTX1vLh6MzPdxWU9QEeOTO6TNAy4G/i1pOdp41HvZnZgTy7fSH1juIvLeoSOHJl8NiK2RsTPyDz2ZCJwZU6qMusFHl2ygTFD+nLSxKFpl2LWaQc8MpH0aeBkYLCkY4C3IuJ3ybpFZJ7TZWYdsLeukaff2sinyydSVOSHSljha08313NAP+BLwM+AoyRtIzNmyN4c1mbWY/3XW5mxS2Yd5y4u6xnaM57JOuDfJa2KiOcAJA0HJgO+NNjsIMxduoFhA/p47BLrMTryOJXnsqa3kBlTxMw6qK6hiceXV3He8WMp8dgl1kP4X7JZN3t+1SZ21jQw63h3cVnPkXqYSBouaZ6kFcn7sFbaTJf0gqSlkhZJ+kzWut9KelvSwuQ1vXu/gVnHzF26gUF9Szhtysi0SzHrMqmHCXAN8ERETCUz1vw1rbTZA/yPiDgOmAX8XFL29ZTfjojpyWth7ks2OziNTcFjS6s48+jR9OtTnHY5Zl0mH8JkNnB7Mn07meGA9xERb0XEimR6PbCRzHPCzArK/He2sHl3na/ish4nH8JkTERUAiTvo/fXWNIMoBRYlbX4hqT76yZJffez7VWSFkhaUF1d3RW1m3XIw4sq6d+nmDOP9u9C1rN0S5hIelzSklZeszu4n3HA74AvRERTsvha4GjgFGA48J22to+IWyOiPCLKR43yf2brXg2NTfxxSSVnHTOaAaUdHkrILK91y7/oiDinrXWSqiSNi4jKJCw2ttFuCPAw8LcR8WLWviuTyVpJvwG+1YWlm3WZl9/ewqZddVxwwri0SzHrcvnQzTUHuCKZvgJ4sGUDSaVkxlP594i4t8W6ccm7yJxvWZLTas0O0kOLKxlQWswZR+23J9esIOVDmNwIzJS0ApiZzCOpXNJtSZtPA38GXNnKJcC/l7QYWAyMBH7cveWbHVhDYxOPLtnAOceMoX+pr+Kynif1jtuI2Ayc3cryBWSeB0ZE3AHc0cb2Z+W0QLMu8MLqzWzZXcfHp7mLy3qmfDgyMevxHl5UyaC+JZx+pC/8sJ7JYWKWY/WNTTy6dAMzjx3jGxWtx3KYmOXYsys3sW1PPef7Ki7rwRwmZjn2h9fWMXRAH3dxWY/mMDHLoV21DcxduoGPnzCO0hL/d7Oey/+6zXJo7pIN1NQ3cfHJ49MuxSynHCZmOfTAa+uYOLw/J5e9b2QFsx7FYWKWI1U7anhu1SY+MX08mQc0mPVcDhOzHHlw4Toi4KKT3MVlPZ/DxCxHHnhtPSdOHMrhowalXYpZzjlMzHLgjQ07WF65g4t9VGK9hMPELAfumV9Bn2JxgZ/FZb2Ew8Ssi9U2NPKfr1Uw89gxjBjU5sCfZj2Kw8Ssi81dWsW2PfVcekpZ2qWYdRuHiVkXu+vld5kwrD8fOWJk2qWYdZvUw0TScEnzJK1I3lu9u0tSY9bAWHOylk+W9FKy/d3JqIxmqVizeTfPr9rMZ8onUlTke0us90g9TIBrgCciYirwRDLfmr0RMT15XZi1/O+Am5LttwJfzG25Zm27a/5aigSXlE9MuxSzbpUPYTIbuD2Zvp3MOO7tkoz7fhZw38Fsb9aV6hubuO+VCs46ejRjD+mXdjlm3SofwmRMRFQCJO+j22jXT9ICSS9Kag6MEcC2iGhI5isAX9hvqXhi+Uaqd9byGZ94t16oW8aAl/Q4MLaVVd/rwG7KImK9pMOBJyUtBna00i72U8dVwFUAZWX+D29d6/bn3+HQQ/px5lEet8R6n24Jk4g4p611kqokjYuISknjgI1t7GN98r5a0tPAScD9wFBJJcnRyQRg/X7quBW4FaC8vLzN0DHrqOWVO3hh9WauOe9oSorz4YDfrHvlw7/6OcAVyfQVwIMtG0gaJqlvMj0S+DCwLCICeAr41P62N8u13z73Dv36FHHpKT7xbr1TPoTJjcBMSSuAmck8ksol3Za0OQZYIOl1MuFxY0QsS9Z9B/iGpJVkzqH8qlurt15v865aHli4jk+ePIGhA3xluvVO3dLNtT8RsRk4u5XlC4AvJdPPAye0sf1qYEYuazTbnztffpe6hia+8OFJaZdilpp8ODIxK1h1DU387sU1/NmRozhi9OC0yzFLjcPErBP+uKSSqh21PiqxXs9hYnaQmpqCXz61iqmjB3H6VF8ObL2bw8TsID22rIo3q3Zy9VlH+Dlc1us5TMwOQkTwz0+uYPLIgVww7dC0yzFLncPE7CA8+cZGlq7fwVfOmEKxj0rMHCZmHRUR/NOTK5k4vD8XeYx3M8BhYtZhf1qxidfXbuMrZxxBHz86xQxwmJh1SFNT8A9z32T80P588uQJaZdjljccJmYd8ODr61i8bjvf+tiRlJb4v49ZM/9vMGunmvpGfvrom5ww/hBmn+hzJWbZHCZm7fSrZ99m/fYavnv+Mb6vxKwFh4lZO2zaVcu/PL2Kc44Zw6lTRqRdjlnecZiYtcP/fewt9tY3cu35R6ddillecpiYHcD8d7Zw58vvcuVpk5gyalDa5ZjlJYeJ2X7U1Ddyzf2LmDCsP98898i0yzHLW6mHiaThkuZJWpG8D2ulzZmSFma9aiRdlKz7raS3s9ZN7/5vYSiKE/UAAAp/SURBVD3VL59ayarq3dzwiRMYUJr6WHJmeSv1MAGuAZ6IiKnAE8n8PiLiqYiYHhHTgbOAPcBjWU2+3bw+IhZ2S9XW472xYQe/fHoVF580ntOP9CPmzfYnH8JkNnB7Mn07cNEB2n8K+GNE7MlpVdar1TY08u17FzGkfx/+9oJj0y7HLO/lQ5iMiYhKgOR99AHaXwrc2WLZDZIWSbpJUt+2NpR0laQFkhZUV1d3rmrr0f7PI2+weN12/s/FJzB8YGna5ZjlvW4JE0mPS1rSymt2B/czDjgBmJu1+FrgaOAUYDjwnba2j4hbI6I8IspHjXK3hbVu7tIN/Pb5d/jChyfxsePGpl2OWUHoljOKEXFOW+skVUkaFxGVSVhs3M+uPg08EBH1WfuuTCZrJf0G+FaXFG290tote/j2va8zbcIhXHveMWmXY1Yw8qGbaw5wRTJ9BfDgftpeRosuriSAkCQy51uW5KBG6wX21jXy1f94lQj4xWUn+0GOZh2QD/9bbgRmSloBzEzmkVQu6bbmRpImAROB/2qx/e8lLQYWAyOBH3dDzdbDNDYFX7vrNRav287PPjOdshED0i7JrKCkfuF8RGwGzm5l+QLgS1nz7wDve1RrRJyVy/qs54sIrn9oGfOWVfGjC49j5rFj0i7JrODkw5GJWap+9ezb/Pb5d/jSRyZzxWmT0i7HrCA5TKxX+/Wzb/Pjh5dz/glj+e75PuFudrBS7+YyS8st/7WKG//4BrOOG8vPP3OSxygx6wSHifU6EcE/PbGSmx5/iz8/8VB+9ukT6VPsg3SzznCYWK9SU9/Itf+5mAdeW8fFJ4/np586kWIfkZh1msPEeo0N22v48u8W8HrFdr4580iuPusIMrcnmVlnOUysV3jqjY38zf2L2FPbwL9+/gN+TIpZF3OYWI+2q7aBGx5exp0vr+WoMYO544sf5Kixg9Muy6zHcZhYjxQRzF1axY8fXsa6bXv58umH842ZR9K3pDjt0sx6JIeJ9TjL1u/g+oeW8cLqzUwdPYh7vnwqp0wannZZZj2aw8R6jMUV2/nl0yt5dOkGhvbvw/Wzj+OyGWWU+LJfs5xzmFhBa2hs4uk3q7n9hXd4ZsUmBvcr4StnTOGqj07hkAF90i7PrNdwmFhBWlG1kwcXrufeV9ZStaOWUYP78jezjuJzHzqMIf0cImbdzWFiBaGhsYnXK7bx1BvV/HFJJauqd1MkOP3IUVw3u4yzjh7tu9jNUuQwsbxUU9/I0vXbeXXNNl56ewsvrd7MztoGigQfOnwEV56WGVJ39JB+aZdqZjhMLGWNTcH6bXtZvWk3K6p2srxyJ29s2MFbVTupbwwAyoYP4IITD+UjR4zktCkjGDawNOWqzayl1MNE0iXAD4FjgBnJoFittZsF/CNQDNwWEc0jMk4G7gKGA68Cn4+Ium4o3Q6grqGJLbvr2Ly7luqdtWzcUcvGnTWs21bDum17qdi6h4qte6lraHpvm1GD+3LMuCF8ceooTi4byvSyoYwe7KMPs3yXepiQGbP9YuBf22ogqRi4mcywvhXAfElzImIZ8HfATRFxl6RbgC8C/5L7sgtLU1PQGEFjU/KKoLExaGgKGpqaaGgM6hubqE/eaxua3nuva2iitqGRmvom9tY3UlvfyJ66RnbXNbCntpFdtQ3srGlgV2092/c2sGNvPdv31rOrtqHVWoYPLGX80P4cNWYw5xwzhsNHDmTyyIFMGT2IkYP6dvOfjJl1hdTDJCKWAwd64N4MYGVErE7a3gXMlrQcOAu4PGl3O5mjnJyFyfceWMxLb295bz4iWm0XbcxE1jbNiyMgkrmIzCt7/5HVJvPe3C6zrimCpqbMuqYImpL3CN4LkFwoLSliQGkxg/uVMKhvHwb1LWb80H4cM24wh/Tvw/ABpQwfVMqIgaWMGtyX0YP7MWpwX/r18V3oZj1N6mHSTuOBtVnzFcAHgRHAtohoyFr+vnHim0m6CrgKoKys7KAKOTT5jXrfHbfxeft+9j7Lm2eVtV5ZDZrnpP9uL5R5V6ZRkaBImWVFUtY0FBUpWQbFEkVFeu+9pEgUN78XF1GSTPcpLqKkWJQWF1FaUkRpcRF9+xTRt6SYviVF9OtTnLyK6N+n2DcDmtl7uiVMJD0OtPaY1u9FxIPt2UUry2I/y1sVEbcCtwKUl5cf1K/rXz3ziIPZzMysR+uWMImIczq5iwpgYtb8BGA9sAkYKqkkOTppXm5mZt2oUPop5gNTJU2WVApcCsyJzMmHp4BPJe2uANpzpGNmZl0o9TCR9AlJFcCpwMOS5ibLD5X0CEBy1HE1MBdYDtwTEUuTXXwH+IaklWTOofyqu7+DmVlvp7auRurpysvLY8GCVm9pMTOzNkh6JSLKWy5P/cjEzMwKn8PEzMw6zWFiZmad5jAxM7NO67Un4CVVA2vSrqODRpK5t6Y38XfuHfydC8dhETGq5cJeGyaFSNKC1q6i6Mn8nXsHf+fC524uMzPrNIeJmZl1msOksNyadgEp8HfuHfydC5zPmZiZWaf5yMTMzDrNYWJmZp3mMClQkr4lKSSNTLuWXJP0U0lvSFok6QFJQ9OuKVckzZL0pqSVkq5Ju55ckzRR0lOSlktaKul/pV1Td5BULOk1SQ+lXUtXcZgUIEkTgZnAu2nX0k3mAcdHxDTgLeDalOvJCUnFwM3AecCxwGWSjk23qpxrAL4ZEccAHwK+2gu+M8D/IjOcRo/hMClMNwF/w36GKO5JIuKxZEwbgBfJjKjZE80AVkbE6oioA+4CZqdcU05FRGVEvJpM7yTzA3Z8ulXllqQJwMeB29KupSs5TAqMpAuBdRHxetq1pOQvgD+mXUSOjAfWZs1X0MN/sGaTNAk4CXgp3Upy7udkfhlsSruQrtQtY8Bbx0h6HBjbyqrvAd8Fzu3einJvf985Ih5M2nyPTLfI77uztm6kVpb1iqNPSYOA+4GvR8SOtOvJFUkXABsj4hVJZ6RdT1dymOShiDinteWSTgAmA69Lgkx3z6uSZkTEhm4sscu19Z2bSboCuAA4O3ruzVEVwMSs+QnA+pRq6TaS+pAJkt9HxH+mXU+OfRi4UNL5QD9giKQ7IuJzKdfVab5psYBJegcoj4hCfPJou0maBfwMOD0iqtOuJ1cklZC5wOBsYB0wH7g8IpamWlgOKfNb0e3Aloj4etr1dKfkyORbEXFB2rV0BZ8zsULwC2AwME/SQkm3pF1QLiQXGVwNzCVzIvqenhwkiQ8DnwfOSv5uFya/tVuB8ZGJmZl1mo9MzMys0xwmZmbWaQ4TMzPrNIeJmZl1msPEzMw6zWFiZmad5jAxM7NOc5iY5QlJ50j6Xdp1mB0Mh4lZ/jgReC3tIswOhsPELH+cCIyV9IykDZL2+/BLs3ziMDHLHycCmyLio8BXgM+mXI9ZuzlMzPJA8hj24cA/JItKgG3pVWTWMQ4Ts/xwLPB6RDSPvjcNWJJiPWYd4jAxyw8nAtlDMU8DFqVUi1mHOUzM8sOJ7Bsex+MjEysgHs/EzMw6zUcmZmbWaQ4TMzPrNIeJmZl1msPEzMw6zWFiZmad5jAxM7NOc5iYmVmn/f/iFRtSyHPGEQAAAABJRU5ErkJggg==\n", "text/plain": [""]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["import numpy as np\n", "import matplotlib.pyplot as plt\n", "N = 100\n", "\n", "def main():\n", " h = np.linspace(-5, 5, N)\n", " tanh = (np.exp(h)-np.exp(-h))/(np.exp(h)+np.exp(-h))\n", "\n", " plt.figure()\n", " plt.plot(h, tanh)\n", " plt.xlabel('$h$')\n", " plt.ylabel('$tanh(h)$')\n", " plt.title('Figure 1.4 Tanh function')\n", " \n", " plt.show()\n", "\n", "if __name__ == '__main__':\n", " main()"]}, {"cell_type": "markdown", "metadata": {}, "source": ["$tanh(x)$ is centred at $0$, which means the probability would be negative if the input value is negative and vice versa. The possibility of having a negative value allows the weight to update better than sigmoid. For sigmoid function that only produces positive values, all weights to the same neuron must either increase together or decrease together. That's a problem, since some of the weights may need to increase while others need to decrease. That can only happen if some of the input activations have different signs. That suggests replacing the sigmoid by an activation function, such as tanh, which allows both positive and negative activations. Of course, the tanh has slightly steeper gradient than the sigmoid but it still faces the vanishing gradient problem."]}, {"cell_type": "markdown", "metadata": {}, "source": ["\n", "\\begin{equation}\n", "ReLU(h)=max(0,h)\n", "\\end{equation}"]}, {"cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [{"data": {"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAEYCAYAAAC9Xlb/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3dd5hU5fnG8e/j0puALCAdARGUjhSJUdEkCLYkRlEwMSYWFJWoMXZNMT2WRGxJ/MXIooBiiR1bLFEElgXpINLbUpa+sOX5/TFDgkiZWXbOmZlzf65rL5idc+Z9Zhbuefc95zxj7o6IiGS3I8IuQEREUk9hLyISAQp7EZEIUNiLiESAwl5EJAIU9iIiEaCwl6SYWSsz22ZmOWHXks7MbImZnVHBfQeY2cL463xeZdd2kHFPNrP5QY0nwVLYy37Fw2pnPHD2fDVz92XuXsfdy9Kgxmpm9my8VjezUxPcr4OZFZvZmINsc4+ZlcSfd5GZ/cfM+lda8Qf3C+Ch+Ov8QqoGib9m7ffcdvcP3L1jqsaTcCns5WDOjgfOnq9VqRzMzKpUYLcPgeHAmiT2GQ1MSWC7ce5eB2gEvAtMSL68CmkNzA5oLIkIhb0kxczaxGeEVeK325rZ+2a21czeMrPRe2bMZnaqma3YZ///Lm/EZ8/PmtkYM9sCXGpmR5jZLWb2uZltMLPxZtZwf7W4+253f8DdPwQS+k3DzIYCRcDbiT5ndy8F8oDmZpa712OdZWYFe838ux5gzH+Y2a/2uv2V12Wv+z4HjgH+Ff+tovq+S0Lx123Pa7zn5/EDM1tmZuvN7Pa9ts0xs9vir+dWM5tmZi3N7P34JjPi41y4b11m1snM3os/v9lmds4+z2m0mb0Sf9zJZtYu0ddUgqewl8M1FvgUOAq4B7gkyf3PBZ4F6hML1OuA84BTgGbAJmIz8cNmZvWILZHcmOR+1YDvAxvi9WBmPYEngCuJPffHgJfMrPrh1Oju7YBl/O+3ql0J7vo1oCNwOnCXmXWKf/8G4CJgMFAPuAzY4e5fj9/fLT7OuL0fzMyqAv8C3gQaA9cCeWa29zLPRcDPgQbAIuDepJ6sBEphLwfzQnxWV2RmX1k7NrNWwInAXfFZ9ofAS0mO8bG7v+Du5e6+k1h43u7uK+JBdw9wfgWXePb1S+Dv7r48we0vMLMiYCdwOXB+fJZP/PZj7j7Z3cvc/UlgF9CvEuqsiJ+7+053nwHMALrFv/9j4A53n+8xM9x9QwKP1w+oA/w2/rN9B3iZWMDvMdHdP93rN5/ulfd0pLIp7OVgznP3+vGv/Z0V0gzY6O479vpeokF6oO1bA8/veZMB5hJbommS5ON+iZl1B84A7k9it/HuXj8+9iyg1z513rjXm2ER0JLYaxKGvY9Z7CAW1BCr6fMKPF4zYLm7l+/1vaVA8wTGlDRUGbMlia7VQEMzq7VX4Lfc6/7tQK09N+Kna+byZfu2XV0OXObuH1VyracCbYBlZgaxYMoxs87u3vNgO7r7ejO7EphiZmPdfXW8znvdPZGliy+9DkDTJGs/nP2XA+2IvVklYxXQ0syO2CvwWwELknwcSROa2UuFuftSYCpwT/w0yP7A2XttsgCoYWZD4mvAdwCHWtN+FLjXzFoDmFmumZ17oI3jBzBrxG9WM7MaFk/zfTxOLPS6x78eBV4BvnXIJwq4+zzgDeDm+Lf+ClxlZn0tpnb8edbdz+4FwGAza2hmTYFRiYy5z/5DzayqmfUGzk9i378Bv7TY6aZmZl3N7Kj4fWuJHQzen8nE3mRujo97KrGf7TNJ1i5pQmEvh2sY0J/YwctfAeOIrV3j7puBq4kFzkpi4bHfs1D28iCxdf83zWwr8AnQ9yDbzye2pt6cWBjvJLbEQvwslNfitexw9zV7voBtQLG7FybxXP8AXGFmjd19KrF1+4eIHbRdBFx6gP2eIraOvoTYAc9xB9juQO4k9ka1idgB0bFJ7HsfMD4+7hbg70DN+H33AE/Gl6Eu2Hsnd98NnAOcCawHHga+H3/Tkwxk+vASqUxmNg6Y5+53h12LiPyPZvZyWMzsRDNrFz8/fhCxUylTdtWniFSMDtDK4WoKTCR2rvkKYIS7Tw+3JBHZl5ZxREQiQMs4IiIRkLbLOI0aNfI2bdqEXYaISMaYNm3aenff91oWII3Dvk2bNkydOjXsMkREMoaZLT3QfVrGERGJAIW9iEgEKOxFRCJAYS8iEgEKexGRCAjsbBwzWwJsJdabvNTdewc1tohI1AV96uVp7r4+4DFFRCJPyzgiImni0y828rcPFpOKNjZBhr0T61E+zcyu2N8GZnaFmU01s6mFhcm0GRcRyWzrthZzzdh88iYvY2dJWaU/fpBhPyD+8W9nAteY2df33cDdH3f33u7eOzd3v1f8iohkndKycq4dO52txSU8MrwntapV/gp7YGHv7qvif64Dngf6BDW2iEg6+9OkBUz+YiO//nYXjmtaLyVjBBL28c/nrLvn78A3Sf4DkEVEss6kOWt55L3PubhvK77Ts0XKxgnqbJwmwPPxz4GuAox199cDGltEJC0t3bCdG8YX0KX5kdx1VueUjhVI2Lv7YqBbEGOJiGSC4pIyRozJ5wgzHh7WkxpVc1I6Xtq2OBYRyWZ3vzibOau38MSlvWnZsFbKx9N59iIiARs/dTnjpi5n5GntGXhck0DGVNiLiARo9qrN3PnCLAa0P4qffOPYwMZV2IuIBGTzzhKuzsunQa1qPDi0BzlHWGBja81eRCQA7s5NE2awctNOxl3Zj0Z1qgc6vmb2IiIBeOz9xUyas5bbBneiV+uGgY+vsBcRSbFPFm/g96/PY0iXo/nhgDah1KCwFxFJoXVbihk5djptGtXmd+d3JX5xaeC0Zi8ikiKlZeWMfHo623eVMvbyvtSpHl7kKuxFRFLkD2/M59MvNvLAhd05tkndUGvRMo6ISAq8MXsNj72/mGF9W3Fej+Zhl6OwFxGpbEvWb+em8TPo2uJI7jo7tQ3OEqWwFxGpRMUlZYzIyycnJ9bgrHqV1DY4S5TW7EVEKtGdL8xi3potPHHpibRokPoGZ4nSzF5EpJKMm7KMCdNWcO1p7TmtY+Owy/kShb2ISCWYtXIzd744m5M7NOL6M4JrcJYohb2IyGHa0+DsqNrVeODC7oE2OEuU1uxFRA5Deblz4/gZrCraybgr+3NUwA3OEqWZvYjIYXj0/c95a+5abh/SiV6tG4RdzgEp7EVEKug/n6/nj2/MZ0jXo7n0pDZhl3NQCnsRkQpYu6WY656eTttGtfndd8NrcJYordmLiCSppKyckWPz2b6rjLGX9wu1wVmi0r9CEZE08/vX5zFlySYeHBp+g7NEaRlHRCQJr89azV8/+IJL+rXm3O7hNzhLlMJeRCRBX6zfzk8nzKRby/rccVansMtJisJeRCQBO3eXMWLMNKqkWYOzRGnNXkTkENydO16Yxfy1W/m/S0+kef2aYZeUNM3sRUQO4Zkpy3kufwXXDezAqWnW4CxRCnsRkYOYtXIzd78Ua3B23ekdwi6nwhT2IiIHsHlHCVeNmUaj2tV4cGiPtGxwliit2YuI7Ed5uXPD+ALWbilm/JX9aVi7WtglHZZAZ/ZmlmNm083s5SDHFRFJ1iP//py3563jjiGd6dEqfRucJSroZZzrgbkBjykikpSPFq3nT2/O5+xuzfh+/9Zhl1MpAgt7M2sBDAH+FtSYIiLJWrM51uDsmNw6/PY7XdK+wVmigpzZPwDcDJQfaAMzu8LMpprZ1MLCwuAqExHhfw3OdpaU8ciwntTOgAZniQok7M3sLGCdu0872Hbu/ri793b33rm5uUGUJiLyX799bR5Tl27it9/tSocMaXCWqKBm9gOAc8xsCfAMMNDMxgQ0tojIIb362Wr+/uEX/KB/a87p1izscipdIGHv7re6ewt3bwMMBd5x9+FBjC0iciiLC7dx87Mz6d6yPrcP6Rx2OSmhi6pEJNJ27C5lxJh8qsYbnFWrkp2xGPjRB3d/D3gv6HFFRPbl7tzx/CwWrNvKPy/rQ7MMbHCWqOx8CxMRScDYT5cxcfpKRp1+LCd3yO6TQhT2IhJJM1cU8fOX5nDKsblcO7B92OWknMJeRCKnaMduRozJJ7dudR64sDtHZHCDs0RlzxUDIiIJKC93Ro0rYN3WYiZcdRINMrzBWaI0sxeRSBn97iLem1/IXWd1pnvL+mGXExiFvYhExocL13PfWws4r3szhvfLjgZniVLYi0gkrCrayXXPTKd9bh1+nUUNzhKlsBeRrLe7NNbgbFdJGY9e0ota1aJ3uDJ6z1hEIuc3r80lf1kRoy/uSbvcOmGXEwrN7EUkq708cxX/99ESfjigDUO6Hh12OaFR2ItI1lq0bhs/e3YmPVvV59YzO4VdTqgU9iKSlbbvKmXEmGlUr5rD6CxucJYordmLSNZxd257/jMWFW7jqcv6cvSR2dvgLFHRfqsTkaw05pOlvFiwihvOOJavdWgUdjlpQWEvIlmlYHkRv3h5Dqd1zOWa07K/wVmiFPYikjU2bd/NNXn5NK5bg/sj0uAsUVqzF5GsUFbuXD+ugMKtu3h2RH/q14pGg7NEaWYvIlnhL+8s5P0Fhdx1dme6tohOg7NEKexFJOP9e0EhD769kG/3aM6wvq3CLictKexFJKOtLNrJqGemc2zjutz77RMi1+AsUQp7EclYu0vLuSYvn5Iy5+HhPSPZ4CxRemVEJGPd+8ocCpYX8fCw6DY4S5Rm9iKSkV6asYonP17KZQPaMrhLdBucJUphLyIZZ+Hardzy3Ex6tW7ArYOPC7ucjKCwF5GMsn1XKSPy8qlZNYfRF/ekao5iLBFasxeRjOHu3DLxMxYXbmPMj/rS9MgaYZeUMfSWKCIZ458fL+VfM1Zx4zc7clJ7NThLhsJeRDJC/rJN/OqVOZx+XGNGnNIu7HIyjsJeRNLexu27GZmXT5N6NbjvAjU4qwit2YtIWisrd65/Zjrrt+9m4oiTOLJW1bBLykiBzOzNrIaZfWpmM8xstpn9PIhxRSTz/fnthXywcD0/P+d4Tmh+ZNjlZKygZva7gIHuvs3MqgIfmtlr7v5JQOOLSAZ6b/46/vzOQr7bswVDT2wZdjkZLZCwd3cHtsVvVo1/eRBji0hmWlm0k1HjCujYpC6/Ok8Nzg5XYAdozSzHzAqAdcAkd5+8n22uMLOpZja1sLAwqNJEJM3sKi3j6rx8ysqcR4b3oma1nLBLyniBhb27l7l7d6AF0MfMTtjPNo+7e293752bmxtUaSKSZu59ZS4zlhfxh+91pW2j2mGXkxUCP/XS3YuA94BBQY8tIunvxYKV/PPjpVx+clsGnaAGZ5UlqLNxcs2sfvzvNYEzgHlBjC0imWPB2q3c8txnnNimATcPUoOzypT0AVozqw0Uu3tZErsdDTxpZjnE3mDGu/vLyY4tItlr265SrhozjdrVq/CQGpxVukOGvZkdAQwFhgEnEjuNsrqZFQKvAo+7+8KDPYa7zwR6HH65IpKN3J2fPTeTJeu3k/fjfjSppwZnlS2Rt853gXbArUBTd2/p7o2Bk4FPgN+a2fAU1igiWe4f/1nCKzNXc9O3OtK/3VFhl5OVElnGOcPdS/b9prtvBJ4DnotfKCUikrRpSzdx7ytzOaNTE676uhqcpcohw35P0JvZQGJLOUXALGAmMMvdd+3vzUBE5FA2bNvFyLH5NKtfkz9d0E0NzlIomQO0Y4Br4vt0Bc4Djgfap6AuEclysQZnBWzY0+CsphYIUimZsF/k7s/H/z4hFcWISHQ8+NYCPly0nt99t4sanAXgkAdozeyfZjYK+NjMbgygJhHJcu/OW8ef31nE93q14MITW4VdTiQkcjbOk/HtmgKXmNlSM3vJzH5pZt9LbXkikm2Wb9zBqHEFdDq6Hr887ytdUyRFEjlA+zbw9p7bZlYF6Ax0A/qiJR0RSdCu0jKuGZtPebnzyLCe1KiqBmdBSeSiKou3KAbA3UuJnYkzE3hqf9uIiOzPL/41h5krNvPYJb1oowZngUrooiozu9bMvrSwZmbVzGygmT0J/CA15YlItnh++gryJi/jyq8fw7eObxp2OZGTyNk4g4DLgKfNrC2x8+xrADnAm8D97l6QuhJFJNPNX7OV2ybOok/bhvz0Wx3DLieSElmzLwYeBh6OXynbCNgZb1UsInJQW4tLGDFmGnVqVOGhi3pQRQ3OQpFU18v4lbKrU1SLiGSZPQ3Olm7cwdgf96WxGpyFJuGwN7MpxA7KfrbnT3fXZweKyAE98dESXv1sDbeeeRx9j1GDszAl8/vUucROs6wGXAUsMbOlKalKRDLetKUb+c2rc/lm5yZc8fVjwi4n8hKe2bv7KmAV8DqAmXUCzk9RXSKSwdZv28XVefk0b1CTP3yvG2ZqcBa2hGf2+5566e5ziTVCExH5r1iDs+kU7SjhkWG91OAsTSRzgHacmbUEviC2bl8MdEpJVSKSse6ftICPFm3g9+d3pXOzemGXI3GJXEH7T6AAuJ3Ygdn6QBegAXBfSqsTkYzyzry1PPTuIoae2JILercMuxzZS6KN0CB2lexbwBvAj4BjgP4pqktEMszyjTsY9UwBxzerxz3naIU33RxuI7Q+qBGaSOQVl5QxIm8aAI8M66UGZ2koqYuq4CuN0ERE+MXLc5i1cgt//X5vWh1VK+xyZD+SORung5k9YWajU1mQiGSWifkrGDt5GVed0o5vdG4SdjlyAMlcVPUUsSWbkwHM7IT4wVsRiah5a7Zw2/Of0bdtQ2765rFhlyMHkUzYH+HurwFlAO4+C9DHzIhE1JbiEkaMyadejar85WI1OEt3yazZr4q3OHaIfWAJUDMlVYlIWnN3bp4wk2Ubd/D05f1oXFcNztJdMmE/Cvgb0NTMfkisz/2slFQlImnt7x9+weuz13D74E70adsw7HIkAcn0xlliZoOA84iddvke8ESK6hKRNDVlyUZ+89o8Bh3flB+f3DbsciRBh1xkM7NLzKzQzFYAF7v7s8ArQHPgP6kuUETSR+HWXVyTl0/LBjX5/fe6qsFZBknkiMpdwGCgB3CMmU0idlZOVWJLOyISAaVl5Vz39HS2FJfwyPBe1KuhBmeZJJFlnG3uPgXAzH4OrAWO1ccSikTLfZMW8PHiDfzxe93odLQanGWaRGb2Tc3sCjM7BWgCrEg26M2spZm9a2ZzzWy2mV1foWpFJBST5qzl4fc+56I+LTm/V4uwy5EKSGRmfzfQFRhGrNtlXTN7C5gOTHf3sQk8Rilwo7vnm1ldYJqZTXL3ORUtXESCsWzDDm4YX8AJzetx99lqcJapEmmE9vjet82sBbHw7wKcCRwy7N19NfEPKnf3rWY2l9gBXoW9SBrb0+DMUIOzTFeRRmgrgBXAqxUZ0MzaEDvYO3k/910BXAHQqlWrfe8WkYDd89JsZq/awt9/0JuWDdXgLJMFen2zmdUBngNGufuWfe9398fdvbe7987NzQ2yNBHZx4Spy3lmynKuPrUdp3dSg7NMF1jYm1lVYkGf5+4TgxpXRJI3Z9UW7nhhFv2POYobvqEGZ9kgkLCP99H5OzDX3fVRhiJpbEtxCVfnTaN+rar8+SI1OMsWQf0UBwCXAAPNrCD+NTigsUUkQe7OTyfMYMWmnYy+uCe5dauHXZJUkqQP0FaEu38I6LpqkTT31w8W88bstdwxpBO926jBWTbR72ciAsDkxRv43evzGdylKT/6mhqcZRuFvYiwbmsxI5+eTuuGtfjdd9XgLBsFsowjIumrtKyca8dOZ2txCU/9qA911eAsKynsRSLuj28uYPIXG7nvgm4c11QNzrKVlnFEIuzN2Wt49N+fc3HfVnynpxqcZTOFvUhELd2wnRsnzKBL8yO566zOYZcjKaawF4mg4pIyRozJ5wgzHh7WUw3OIkBr9iIRdNeLs5izegtPXKoGZ1Ghmb1IxIyfspzxU1cw8rT2DDxODc6iQmEvEiGzV23mzhdnMaD9UfxEDc4iRWEvEhGbd5ZwdV4+DWpV48GhPcg5QhdORYnW7EUiwN25acIMVm7aybgr+9GojhqcRY1m9iIR8Nj7i5k0Zy23De5Er9ZqcBZFCnuRLPfJ4g38/vV5DOl6ND8c0CbsciQkCnuRLLZuSzEjx06nTaPaanAWcVqzF8lSpWXljHx6Ott3lZL3477Uqa7/7lGmn75IlvrDG/P59IuNPHBhdzo2rRt2ORIyLeOIZKHXZ63hsfcXM7xfK87r0TzsciQNKOxFssyS9dv56YQZdGtxJHeqwZnEKexFssjO3WVcNWYaOTnG6GE9qV5FDc4kRmv2IlnC3bnzxVnMX7uVJy49kRYN1OBM/kcze5EsMW7Kcp6dtoJrT2vPaR0bh12OpBmFvUgWmLVyM3e9NJuTOzTi+jPU4Ey+SmEvkuE27yjhqjHTOKp2NR64sLsanMl+ac1eJIOVlzs3jC9g7ZZixl3Zn6PU4EwOQDN7kQz2yL8/5+1567h9cCd6tmoQdjmSxhT2IhnqP5+v509vzufsbs34wUltwi5H0pzCXiQDrdlczHVPT6dto9r85jtd1OBMDklr9iIZpqSsnJFj89mxu4ynL++nBmeSEP0rEckwv3ttHlOXbuLBod3p0EQNziQxgSzjmNkTZrbOzGYFMZ5Itnrts9X87cMv+H7/1pzbXQ3OJHFBrdn/AxgU0FgiWWlx4TZ++uxMurWsz+1DOoVdjmSYQMLe3d8HNgYxlkg22rm7jBFj8qmaYzysBmdSAWl1No6ZXWFmU81samFhYdjliKQFd+f2Fz5jwbqtPDC0B83r1wy7JMlAaRX27v64u/d29965ublhlyOSFp7+dDkT81dy3cAOnHKs/l9IxaRV2IvIl81cUcQ98QZn153eIexyJIMp7EXSVNGO3YwYk0+jOtV4cGgPNTiTwxLUqZdPAx8DHc1shZn9KIhxRTJVebnzk3EFrNtazMPDe9GwdrWwS5IMF8hFVe5+URDjiGSLh99bxLvzC/nFucfTvWX9sMuRLKBlHJE089Gi9dw3aQHndGvGJf1ah12OZAmFvUga2dPg7JjcOmpwJpVKvXFE0kRJWTnXjM1nZ0kZ44b3pLYanEkl0r8mkTTxm1fnMW3pJv5yUQ/aN1aDM6lcWsYRSQMvz1zFEx99waUnteHsbs3CLkeykMJeJGSL1m3jZ8/OpEer+tw2WA3OJDUU9iIh2rG7lKvzplG9ag6jL+5JtSr6LympoTV7kZC4O7dN/IyF67bxz8v60EwNziSFNI0QCcmYyct4oWAVo04/lpM7qMGZpJbCXiQEM5YX8ct/zeHUjrlcO7B92OVIBCjsRQK2afturs7LJ7dude6/oDtHqMGZBEBr9iIBKi93fjK+gMKtu5hwVX8aqMGZBEQze5EAPfTuIt6bX8hdZ3emmxqcSYAU9iIB+WBhIfe/tYBv92jOsL6twi5HIkZhLxKAVUU7uf6ZAjo0rsO93z5BDc4kcAp7kRTbXRprcLa7tJxHhveiVjUdKpPg6V+dSIr9+tW5TF9WxOiLe9Iut07Y5UhEaWYvkkIvzVjFP/6zhMsGtGVI16PDLkciTGEvkiKL1m3lludm0qt1A24dfFzY5UjEKexFUmD7rlJGjMmnZrzBWdUc/VeTcGnNXqSSuTu3TvyMzwu38dSP+tL0yBphlySimb1IZXvqk6W8NGMVN3zjWAa0bxR2OSKAwl6kUk1ftolfvjyHgcc15upT1eBM0ofCXqSSbNy+m2vy8mlSrwb3XdBNDc4krWjNXqQSlJU7o8YVsH7bbp4bcRL1a6nBmaQXhb1IJfjLOwt5f0Ehv/52F7q0ODLsckS+Qss4Iofp3wsKefDthXynZ3Mu6tMy7HJE9kthL3IYVhXtZNQz0+nYpC73ntdFDc4kbSnsRSpod2k5V+flU1LmPDysJzWr5YRdksgBac1epILufWUOBcuLeHR4T45RgzNJc5rZi1TAiwUrefLjpfz4a20ZdIIanEn6CyzszWyQmc03s0VmdktQ44pUttdnrebWiZ9xYpsG/OxMNTiTzBDIMo6Z5QCjgW8AK4ApZvaSu88JYnyRyrBuazF3vzib12at4fhm9XhIDc4kgwS1Zt8HWOTuiwHM7BngXKDSw/7sv3xIcUlZZT+sCKs3F7O7rJybB3Xk8pOPUdBLRgkq7JsDy/e6vQLou+9GZnYFcAVAq1YV+0Dmdrm12V1WXqF9RQ6me8v6XHlKO9o31sFYyTxBhf3+Tj72r3zD/XHgcYDevXt/5f5EPDC0R0V2ExHJakH9HroC2PvSwhbAqoDGFhGJvKDCfgrQwczamlk1YCjwUkBji4hEXiDLOO5eamYjgTeAHOAJd58dxNgiIhLgFbTu/irwalDjiYjI/+jcMRGRCFDYi4hEgMJeRCQCFPYiIhFg7hW6dinlzKwQWBp2HUlqBKwPu4iA6TlHg55zZmjt7rn7uyNtwz4TmdlUd+8ddh1B0nOOBj3nzKdlHBGRCFDYi4hEgMK+cj0edgEh0HOOBj3nDKc1exGRCNDMXkQkAhT2IiIRoLBPETO7yczczBqFXUuqmdkfzGyemc00s+fNrH7YNaWCmQ0ys/lmtsjMbgm7nlQzs5Zm9q6ZzTWz2WZ2fdg1BcXMcsxsupm9HHYtlUVhnwJm1pLYh6svC7uWgEwCTnD3rsAC4NaQ66l0ZpYDjAbOBDoDF5lZ53CrSrlS4EZ37wT0A66JwHPe43pgbthFVCaFfWrcD9zMfj56MRu5+5vuXhq/+QmxTyLLNn2ARe6+2N13A88A54ZcU0q5+2p3z4//fSux8GseblWpZ2YtgCHA38KupTIp7CuZmZ0DrHT3GWHXEpLLgNfCLiIFmgPL97q9gggE3x5m1gboAUwOt5JAPEBsslYediGVKbAPL8kmZvYW0HQ/d90O3AZ8M9iKUu9gz9ndX4xvczuxX/3zgqwtILaf70XiNzczqwM8B4xy9y1h15NKZnYWsM7dp5nZqWHXU5kU9hXg7mfs7/tm1gVoC8wwM4gtZ+SbWR93XxNgiZXuQM95DzP7AXAWcLpn58UbK4CWe91uAawKqZbAmFlVYkGf5+4Tw8KFRSAAAAFySURBVK4nAAOAc8xsMFADqGdmY9x9eMh1HTZdVJVCZrYE6O3umdY5LylmNgi4DzjF3QvDricVzKwKsYPPpwMrgSnAxdn8WcoWm7E8CWx091Fh1xO0+Mz+Jnc/K+xaKoPW7KUyPATUBSaZWYGZPRp2QZUtfgB6JPAGsQOV47M56OMGAJcAA+M/14L4jFcykGb2IiIRoJm9iEgEKOxFRCJAYS8iEgEKexGRCFDYi4hEgMJeRCQCFPYiIhGgsBdJkJmdYWZPhV2HSEUo7EUS1w2YHnYRIhWhsBdJXDegqZl9YGZrzOygzeFE0onCXiRx3YD17n4ycDUwLOR6RBKmsBdJQLzVb0Pgj/FvVQGKwqtIJDkKe5HEdAZmuPueTy/qCswKsR6RpCjsRRLTDdj7oya7AjNDqkUkaQp7kcR048vhfgKa2UsGUT97EZEI0MxeRCQCFPYiIhGgsBcRiQCFvYhIBCjsRUQiQGEvIhIBCnsRkQj4fxRAGeEsVEzvAAAAAElFTkSuQmCC\n", "text/plain": [""]}, "metadata": {"needs_background": "light"}, "output_type": "display_data"}], "source": ["#ReLu function\n", "def relu(X):\n", " return np.maximum(0,X)\n", "\n", "N = 100\n", "\n", "def main():\n", " h = np.linspace(-5, 5, N)\n", " Relu = relu(h)\n", "\n", " plt.figure()\n", " plt.plot(h, Relu)\n", " plt.xlabel('$h$')\n", " plt.ylabel('$Relu(h)$')\n", " plt.title('Figure 1.4 Relu function')\n", " \n", " plt.show()\n", "\n", "if __name__ == '__main__':\n", " main()"]}, {"cell_type": "markdown", "metadata": {}, "source": ["This activation function is wildly used in CNN (Convolutional Neural Network) because of the two characters: it is easy to compute and it does not have a vanishing gradient problem at all. Nevertheless, it has the biggest problem, which is there is no derivative at the point ($x=0$). We can avoid this by keep our learning rate low. On the other hand, when the weighted input to a rectified linear unit is negative, the gradient vanishes, and so the neuron stops learning entirely. Some recent work on image recognition has found considerable benefit in using rectified linear units through much of the network. However, as with tanh neurons, we do not yet have a really deep understanding of when, exactly, rectified linear units are preferable, nor why."]}, {"cell_type": "markdown", "metadata": {}, "source": ["The following figure 16.1 is a MLP/DNN model with modified activation functions. In this model, the activation functions are changed to ReLU from sigmoid for all hidden layers but the output layer (in order to predict the probability)."]}, {"cell_type": "markdown", "metadata": {}, "source": ["\n", "